6.5. Platform Gateway

How is agentgateway deployed?

Same gateway, same three listeners as Chapter 5 — now a pod instead of a host process. What is new on this page is the network policy around it and where its credentials live.

The base pins agentgateway v1.4.1 by image digest and runs it as UID/GID 65532 with a read-only root and dropped capabilities. It mounts the selected config read-only and exposes one ClusterIP service (one address reachable only from inside the cluster):

agentgateway:3000   MCP
agentgateway:3001   A2A
agentgateway:4000   OpenAI-compatible model
agentgateway:15020  internal metrics

Readiness/liveness probes check the MCP listener. Resource bounds keep the lab schedulable on a small node.

How is the profile selected?

The base does not hard-code a gateway ConfigMap. Each overlay adds one generated ConfigMap:

• Local includes agentgateway/k3d and uses Ollama at host.k3d.internal:11434.
• GKE includes agentgateway/gke and uses Vertex with backendAuth.gcp.

The Python agent still points at agentgateway:3000 and agentgateway:4000; provider movement is a data-plane change.

How does network policy reduce reachability?

A NetworkPolicy is a Kubernetes rule naming which pods may talk to which, on which ports (0.7. Glossary). Ingress rules admit only declared callers:

• The agentops namespace can reach all gateway listeners; kagent can reach only MCP :3000 and model :4000, not A2A or metrics.
• The raw MCP server accepts only gateway pods.
• MLflow accepts only the OTel collector.
• OTel accepts signals from agentops and kagent namespaces.

Egress is denied by default for every pod in the namespace, then reopened one declared flow at a time in network-policies.yaml:

spec:
  podSelector: {}
  policyTypes: [Egress]

A namespace-wide policy allows DNS to kube-system. The base then reopens these flows, and nothing else:

• The gateway reaches the raw MCP server (:8000), the agent A2A port (:8080), and the collector (:4317).
• The agent reaches only gateway :3000/:4000 and collector :4318.
• The collector reaches MLflow :5000, Loki :3100, and gateway metrics :15020.
• The raw MCP server, MLflow, and Loki get no base egress beyond DNS.

The model upstream differs per environment, so each overlay appends its own exception to agentgateway-egress.

The base is a directed allow-list, but each model overlay adds a port-scoped public exception. Solid in-cluster edges below are the only internal flows; the dashed edge is the residual public reach. Note that even the agent cannot reach the raw MCP server directly — it must go through the gateway.

flowchart LR
    kagent["kagent ns"] -->|":3000, :4000"| GW["agentgateway"]
    AG["agentops-agent"] -->|":3000, :4000"| GW
    AG -->|":4318"| OT["otel-collector"]
    GW -->|":8000"| MCP["raw MCP"]
    GW -->|":8080"| A2A["agent A2A"]
    GW -->|":4317"| OT
    OT -->|":5000"| ML["MLflow"]
    OT -->|":3100"| LK["Loki"]
    OT -->|":15020"| GW
    GW -. "local: any IPv4 :11434<br/>GKE: any IPv4 :443" .-> Public["intended model API<br/>destination not enforced"]
    AG -.blocked.-x MCP
    MCP -.blocked.-x ML

The precise source → destination allow-matrix, mirroring network-policies.yaml:

* The model exception is on the gateway pod only, but vanilla NetworkPolicy scopes it by port rather than hostname: any IPv4 :11434 locally, or any IPv4 :443 on GKE, plus WIF :987/:988. MLflow receives the same broad HTTPS exception for intended GCS use. Every pod also keeps a namespace-wide DNS allowance to kube-system.

The broad HTTPS rule is a residual exfiltration path, not Vertex destination allowlisting. Workload Identity IAM limits which Google operations accept the pod identity; it cannot stop a compromised process from sending ordinary HTTPS elsewhere. Production needs an egress proxy, firewall, or FQDN-aware network policy, then deployment of the exact destinations that policy permits.

These policies depend on a network-policy-capable cluster: k3s enforces them out of the box, and the OpenTofu module enables Calico (the plugin that actually enforces the rules) on GKE. They do not cover the kagent control-plane namespace or cross-cluster security.

Why should an agent pod have egress rules?

Assume the model has been talked into leaking your data. Egress rules decide whether it can.

Prompt injection turns a tool-using agent into a confused deputy: a trusted process made to act for an attacker. Hostile text inside an incident, log line, or runbook can instruct the model to send whatever is in its context somewhere else. 4.6. Security hardens the application layer; default-deny-egress blocks undeclared destinations when those layers fail.

A hijacked agent pod can still send data over its allowed model, tool, telemetry, and DNS routes. It cannot open a direct connection to an arbitrary attacker-controlled host.

The same reasoning keeps the model-upstream exception on the gateway pod only, scoped to one port, instead of a namespace-wide allow that every compromised workload would inherit. Port scoping reduces reach; it does not authenticate the destination.

What do the ResourceQuota and LimitRange protect against?

The shared local cluster hosts other projects, so resource-quota.yaml caps what the agentops namespace can claim:

hard:
  requests.cpu: "2"
  requests.memory: 4Gi
  limits.cpu: "8"
  limits.memory: 9Gi
  pods: "12"

The numbers are the declared per-container requests/limits of the deployed workloads plus one surge pod per rolling deployment, so Skaffold rollouts never deadlock against the quota. A runaway or duplicated workload is rejected at admission — when the API server first accepts the object — instead of starving neighbor namespaces.

A compute quota also rejects any pod that declares no resources at all. The LimitRange covers that case: it supplies bounded defaults, so ad-hoc diagnostic pods remain schedulable. Inspect live consumption with:

kubectl -n agentops describe resourcequota agentops-compute

How do you debug a NetworkPolicy lockout?

Egress policies are easy to over-tighten, so practice a deliberate lockout. First verify the guardrail works. The raw MCP server has no egress allowance beyond DNS, so a connection from it to MLflow must time out, even though the collector keeps delivering traces to that same endpoint:

kubectl -n agentops exec deploy/agentops-mcp -- python -c \
  'import urllib.request; urllib.request.urlopen("http://mlflow:5000/health", timeout=5)'

Expect TimeoutError — the policy, not the service, refused the flow.

Now cause a real lockout:

kubectl -n agentops delete networkpolicy dns-egress
kubectl -n agentops exec deploy/agentops-mcp -- python -c \
  'import socket; print(socket.gethostbyname("agentgateway"))'

The probe raises socket.gaierror instead of printing an address: name resolution fails for every pod in the namespace. Restore the declared state by reapplying only the rendered policies:

kubectl kustomize infra/k8s/overlays/local | yq 'select(.kind == "NetworkPolicy")' | kubectl apply -f -

Render the overlay rather than the base, so the overlay's Ollama exception is preserved.

You have now seen both halves of a lockout: the symptom and the restore. When a policy blocks something for real, diagnose in this order:

1. List which policies select the failing pod: kubectl -n agentops get networkpolicy and kubectl -n agentops describe networkpolicy default-deny-egress.
2. Separate resolution from connection: retry the socket.gethostbyname probe, then the urllib probe against an allowed destination.
3. Check that one selected policy allows the destination's port and peer; default-deny plus no matching allow is a lockout.

How do you access the private services?

No Ingress or LoadBalancer is created. Forward agentgateway in one terminal:

kubectl -n agentops port-forward svc/agentgateway \
  3000:3000 3001:3001 4000:4000 15020:15020

Forward MLflow from a second terminal:

kubectl -n agentops port-forward svc/mlflow 5000:5000

Leave both forwards running while you use them. Use http://localhost:3001 for A2A clients. Direct agent port 8080 is a raw bypass for diagnostics, not a capability-limited diagnosis surface. The model listener enforces the demo API key from the agentgateway-client Secret, so a forwarded call to :4000 needs -H "Authorization: Bearer agentgateway" (5.5. Gateway Security).

How does GKE authenticate to Vertex?

On GKE, no key file exists anywhere. The gateway pod borrows a Google identity with two narrow roles, and Google hands it short-lived credentials on demand.

How do you keep secrets in git safely?

Commit ciphertext, never plaintext, and keep the decryption key out of git.

The course platform holds no real credential today, so nothing below is required to finish the chapter. Read it the day you do hold one, such as a hosted-model API key or a Grafana admin password. The worked example pairs SOPS, which encrypts only the values inside a YAML file, with age keys.

infra/scripts/secrets.sh keygen                  # prints your public recipient
# put that recipient in .sops.yaml, write your Secret manifest, then:
infra/scripts/secrets.sh encrypt infra/k8s/base/secrets/my-secret.enc.yaml
infra/scripts/secrets.sh decrypt infra/k8s/base/secrets/my-secret.enc.yaml

infra/scripts/secrets.sh decrypt \
  infra/k8s/base/secrets/agentgateway-client.enc.yaml | kubectl apply -f -

What must never be committed?

These three rules hold whether or not you ever use SOPS:

1. The age private key. infra/secrets/, **/*.agekey, and any decrypted **/*.dec.yaml output are gitignored; verify with git check-ignore -v infra/secrets/age.agekey. Publishing the private key retroactively decrypts every ciphertext in git history.
2. Plaintext Secret manifests. scripts/check-infra.sh fails if any file under infra/**/secrets/ lacks SOPS metadata or carries a data/stringData value that is not ENC[...] ciphertext, and the repository gitleaks/trivy secret scans stay green because SOPS ciphertext contains no recognizable credential.
3. Real values as "temporary" placeholders. The only committed plaintext key-like value in the platform is the agentgateway marker, which the gateway enforces as a demo API key (Chapter 5.5); anything with actual authority goes through the encrypted path or, better, never exists thanks to Workload Identity.

If a private key does leak, treat every secret encrypted to it as compromised: rotate the credentials themselves, not just the key — git history preserves the old ciphertext forever.

How would you narrow one ingress rule to least privilege?

Optional exercise: diagnose why the quarantined collector-ingress fixture is broader than the completed reference.

• Mode: inspect.
• Goal: explain why namespace-only selectors plus ports 4317, 4318, and 8889 admit more senders than the source-and-port-specific completed policy.
• Files to touch: none. Read infra/k8s/exercises/otel-ingress-broad.yaml, infra/k8s/base/network-policies.yaml, and infra/k8s/overlays/local/network-policies.yaml.
• Preflight: require git diff --quiet -- infra/k8s/exercises/otel-ingress-broad.yaml infra/k8s/base/network-policies.yaml infra/k8s/overlays/local/network-policies.yaml so the comparison uses the reviewed fixtures.
• Gate that proves completion: mise run check:infra exits zero. It verifies the broad fixture's exact unsafe shape, proves neither overlay includes it, and asserts that the completed base permits only named OTLP emitters while only local Prometheus reaches 8889.
• Final state: no file or cluster changes; the focused git diff --quiet -- command still exits zero.

What proves this page worked?

With the local profile, call all three gateway ports through one port-forward and inspect :15020/metrics. Verify no service has type LoadBalancer or NodePort:

kubectl -n agentops get svc -o custom-columns=NAME:.metadata.name,TYPE:.spec.type

Then open one forward and leave it running:

kubectl -n agentops port-forward svc/agentgateway \
  3000:3000 3001:3001 4000:4000 15020:15020

In a second terminal, call the A2A listener, the model listener, and the metrics endpoint:

curl -fsS http://localhost:3001/.well-known/agent-card.json | jq .name
curl -fsS http://localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer agentgateway' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b-instruct",
    "messages": [{"role": "user", "content": "Reply with exactly: gateway ready"}],
    "temperature": 0
  }' | jq -r '.choices[0].message.content'
curl -fsS http://localhost:15020/metrics | head

The optional GKE profile needs a stronger check because a plain answer never exercises tool-result translation. This command makes billed Vertex calls, so run it only inside an explicitly approved cloud lab:

mise run gke:smoke

The task verifies the exact OpenTofu-derived context, the HEAD-tagged Agent image, and all three live model owners. It opens random loopback-only forwards for its run and closes them afterward.

The smoke first echoes a synthetic function result through :4000. It then requires the A2A agent to reach a completed task and return INC-002's stable ID, severity, and title. An exact one-call gateway-counter delta proves get_incident ran once. Both turns are read-only, and the proof remains valid if the incident's runtime status changed earlier.

MCP :3000 needs an MCP client rather than a plain curl. Run the list-tools script from 5.2. MCP Gateway against http://127.0.0.1:3000/mcp, exactly as 6.4. Platform Tools does.

You are done when:

• Every row of the service list reads ClusterIP: no LoadBalancer, no NodePort.
• The :3001 call prints the agent card's name, AgentOps Agent.
• The :4000 call prints a line of model text, so the agentgateway marker was accepted and Ollama answered through the gateway.
• On an approved GKE lab, mise run gke:smoke prints that its exact-source model tool loop and read-only A2A retrieval passed.
• :15020/metrics prints Prometheus metric lines instead of an empty body.
• The MCP-to-MLflow probe still ends in TimeoutError, and DNS resolves again after the restore command.
• You can name, for any arrow in the diagram above, the policy that allows it.

Continue to 6.6. Platform Delivery when the only way you reached any of those ports was a port-forward you started yourself.